How Consistent is Relevance Feedback in Exploratory Search?

Search activities involving knowledge acquisition, investigation and synthesis are collectively known as exploratory search. Exploratory search is challenging for users, who may be unable to formulate search queries, have ill-defined search goals or may even struggle to understand search results. To ameliorate these difficulties, reinforcement learning-based information retrieval systems were developed to provide adaptive support to users. Reinforcement learning is used to build a model of user intent based on relevance feedback provided by the user. But how reliable is relevance feedback in this context? To answer this question, we developed a novel permutation-based metric for scoring the consistency of relevance feedback. We used this metric to perform a retrospective analysis of interaction data from lookup and exploratory search experiments. Our analysis shows that for lookup search relevance judgments are highly consistent, supporting previous findings that relevance feedback improves retrieval performance. For exploratory search, however, the distribution of consistency scores shows considerable inconsistency.Peer reviewe

The Tower of Babel : Holocaust testimonies and the ethics of translation

This article considers Holocaust testimonies and the question of translation, understood here as both exchanges between languages within a text and renditions of a text into another language. According to Imre Kertész, Holocaust has no language that could express its meaning, and no national language has been able to coin words and expressions capable of conveying its catastrophic dimension. Since Holocaust survivors must express themselves in one of the national languages, Holocaust testimony is always a form of translation, even in the case of writers who wrote their memoirs in their native tongues (such as Kertész, Primo Levi, Jean Améry, Paul Celan, Ida Fink, and Hanna Krall, whose work is discussed here). The choice of language in which survivors’ memoirs (as well as other literary forms) were written had a profound impact on their authors’ sense of self-identity, their ability to heal, and the way they remembered the past. The largest number of memoirs appeared in English, the survivors’ second tongue, whose neutrality enabled them to overcome associations with the language in which they experienced traumatic events. Others, such as Elie Wiesel and Isabella Leitner, translated their initial accounts written in their native tongues (Yiddish and Hungarian, respectively) into smoothed-out versions in the languages of their adopted country (France and the United States). The article examines selected instances of important translatory exchanges taking place in Holocaust testimonies. Some of them (Primo Levi’s narratives in particular) demonstrate that, during the Holocaust, translation was a crucial survival strategy, allowing the victim to navigate the incomprehensible “Babel” of the events. Other works, however, such as translation sequences in Claude Lanzmann’s film Shoah or Hanna Krall’s story of Izolda Regensberg (in Król kier znów na wylocie), disclose a failure and treachery of translation. The study employs Emmanuel Levinas’ ethical conception of language and Walter Benjamin’s reflection on translation in “The Task of the Translator” (both thinkers were also translators and their lives were profoundly affected by the Holocaust). Drawing attention to an affinity between Benjamin’s conception of “pure language” and Levinas’ “Saying”, it concludes that – considering the centrality of translation in Holocaust testimony – translation should be acknowledged as a modality of bearing witness in its own right. While Holocaust translations reveal the abyssal, Babelian condition of post-Holocaust speech, they also hope for the renewal of communication and for the tikkun olam of language

How Relevance Feedback is Framed Affects User Experience, but not Behaviour

Retrieval systems based on machine learning require both positive and negative examples to perform inference, which is usually obtained through relevance feedback. Unfortunately, explicit negative relevance feedback is thought to have poor user experience. Instead, systems typically rely on implicit negative feedback. In this study, we confirm that, in the case of binary relevance feedback, users prefer giving positive feedback ( and implicit negative feedback) over negative feedback ( and implicit positive feedback). These two feedback mechanisms are functionally equivalent, capturing the same information from the user, but differ in how they are framed. Despite users' preference for positive feedback, there were no significant differences in behaviour. As users were not shown how feedback influenced search results, we hypothesise that previously reported results could, at least in part, be due to cognitive biases related to user perception of negative feedback.Peer reviewe

Can Language Models Identify Wikipedia Articles with Readability and Style Issues?

Wikipedia is frequently criticised for having poor readability and style issues. In this article, we investigate using GPT-2, a neural language model, to identify poorly written text in Wikipedia by ranking documents by their perplexity. We evaluated the properties of this ranking using human assessments of text quality, including readability, narrativity and language use. We demonstrate that GPT-2 perplexity scores correlate moderately to strongly with narrativity, but only weakly with reading comprehension scores. Importantly, the model reflects even small improvements to text as would be seen in Wikipedia edits. We conclude by highlighting that Wikipedia's featured articles counter-intuitively contain text with the highest perplexity scores. However, these examples highlight many of the complexities that need to be resolved for such an approach to be used in practice.Peer reviewe

Query Suggestions as Summarization in Exploratory Search

Peer reviewe

Statistically Significant Detection of Semantic Shifts using Contextual Word Embeddings

Detecting lexical semantic change in smaller data sets, e.g. in historical linguistics and digital humanities, is challenging due to a lack of statistical power. This issue is exacerbated by non-contextual embedding models that produce one embedding per word and, therefore, mask the variability present in the data. In this article, we propose an approach to estimate semantic shift by combining contextual word embeddings with permutation-based statistical tests. We use the false discovery rate procedure to address the large number of hypothesis tests being conducted simultaneously. We demonstrate the performance of this approach in simulation where it achieves consistently high precision by suppressing false positives. We additionally analyze real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus. We show that by taking sample variation into account, we can improve the robustness of individual semantic shift estimates without degrading overall performance.Peer reviewe

Can models of author intention support quality assessment of content?

Academics seek to find, understand and critically review the work of other researchers through published scientific articles. In recent years, the volume of available information has significantly increased, partly due to technological advancements and partly due to pressures on academics to 'publish or perish'. This amount of papers presents a challenge not only for the peer-review process but also for readers, particularly inexperienced readers, to find publications of high quality. Whilst one might rely on citation or journal rankings to help guide this decision, this approach may not be completely reliable due to biased peer-review processes and the fact that the citation count of an article does not per se indicate its quality. Here, we analyse how expected author intentions in a Related Work section can be used to indicate its quality. We show that author intentions can predict the quality with reasonable accuracy and propose that similar approaches could be used in other sections to provide an overall picture of quality. This approach could be useful in supporting peer-review processes and for a reader in prioritising articles to read. © 2019 CEUR-WS. All rights reserved.Peer reviewe